Method and apparatus for motion compensation prediction

ABSTRACT

The invention relates to a motion compensation performed under an inter-frame prediction. A fractional sample interpolation is applied on retrieved samples to generate fractional samples. The fractional sample interpolation comprises a plurality of filtering operations, and all of filtered results from the filtering operations are truncated down to a predetermined bit depth independent of the bit depth of the samples stored in the reference picture memory.

RELATED APPLICATIONS

The present patent document claims the benefit of the filing date under35 U.S.C. §119(e) of Provisional U.S. Pat. No. 61/501,112, filed Jun.24, 2011, the entire content of which is incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a motion compensation for aninter-frame prediction and in particular to a fractional sampleinterpolation used in the motion compensation which and achieves lowcomplexity and high precision.

2. Description of the Related Art

Digital video requires a large amount of data to represent each andevery frame of a digital video sequence (e.g., series of frames) in anuncompressed manner. It is not feasible for most applications totransmit uncompressed digital video across computer networks because ofbandwidth limitations. In addition, uncompressed digital video requiresa large amount of storage space. The digital video is normally encodedin some manner to reduce the storage requirements and reduce thebandwidth requirements.

One technique for encoding digital video is inter-frame prediction, orinter-prediction. Inter-prediction exploits temporal redundancies amongdifferent frames. Temporally adjacent frames of video typically includeblocks of pixels, which remain substantially the same. During theencoding process, a motion vector interrelates the movement of a blockof pixels in one frame to a block of similar pixels in another frame.Accordingly, the system is not required to encode the block of pixelstwice, but rather encodes the block of pixels once and provides a motionvector to predict the other block of pixels.

Another technique for encoding digital video is intra-frame predictionor intra-prediction. Intra-prediction encodes a frame or a portionthereof without reference to pixels in other frames. Intra-predictionexploits spatial redundancies among blocks of pixels within a frame.Because spatially adjacent blocks of pixels generally have similarattributes, the efficiency of the coding process is improved byreferencing the spatial correlation between adjacent blocks. Thiscorrelation may be exploited by prediction of a target block based onprediction modes used in adjacent blocks.

In the inter-prediction, a received picture is predicted, based onmotion estimation and compensation. Moving objects in video often appearfrom frame to frame with which all or part of it are relocated in thosesubsequent frames. Despite those relocations, correlation among thesequence of the frames is high and gives rise to redundancy. Thistemporal redundancy can be reduced by comparing and relating the samplesin the current frame to the location of the same object in the referenceframes. Specifically, during motion estimation, the current frame or apartition thereof is compared with reference frames which may betemporally previous or forward of the current frame. A pattern of pixelswithin a search range set in the respective reference frame is comparedwith the pattern of pixels exhibited in the current frame until areference frame is found which contains a pixel pattern best matchingthe pixel pattern in the current frame to be encoded. Based on thecomparison results, an inter-frame displacement vector or a motionvector is estimated. Using the estimated motion vector, motioncompensation yields a prediction of the current frame.

The motion vector accuracy and coding efficiency can be increased byapplying interpolation to the pixels in the reference picture, which arecalled samples at integer positions, or simply integer samples, toincrease the resolution of the reference picture. Interpolation is togenerate fractional samples between each integer sample, using thevalues of the integer samples. The more fractional samples are generatedbetween the integer samples, the higher the resolution of the referencepicture becomes, and the more precisely and accurately a fractionalsample displacement can be compensated. For example, in order toaccurately compensate a movement of a moving object which is adisplacement of only half a pixel, at least half-pixel (pel)interpolation is needed. Motion estimation and compensation may beperformed using a number of different block sizes. Individual motionvectors may be determined for partitions having 4×4, 4×8, 8×4, 8×8,8×16, 16×8 or 16×16 pixels. The provision of small motion compensationpartitions improves the ability to handle fine motion details.

H.264/AVC takes a 2-step approach and achieves motion compensation up toa quarter-pel resolution. In H.264/AVC, the first step uses a 6-tapfilter to generate intermediate values at a half-pel resolution from thevalues of surrounding integer samples. In the second step, the values ofinteger samples and the intermediate values are averaged or theintermediate values are averaged among themselves to generate fractionalsamples at quarter-pel positions, or simply quarter-pel samples. In Bslices, two predictions fractional samples from two predictions mayfurther be averaged. Please note, however, that multiple averagingoperations, when cascaded, introduce rounding errors which adverselyaffects the accuracy and the efficiency of motion compensation.Proposals D321 and E242 of Joint Collaborative Team on Video Coding(JCT-VC) address the rounding error issue associated with bi-directionalaveraging. These documents propose that a rounding operation be limitedto taking place at the last step of bi-directional averaging after twopredictions are added.

JCT-VC Draft E603 discloses the use of an 8-tap filter to achieve thequarter-pel resolution. In E603, some of the quarter-pel samples arederived by applying an 8-tap filter to the nearest integer samples andtruncating the filtered results down to a predetermined bit depth. Therest of the quarter-pel samples are derived through two processes. Inthe first process, intermediate values are derived by applying the 8-tapfilter to the nearest integer samples in the vertical direction. In thesecond process, the 8-tap filter is applied to the intermediate valuesin the horizontal direction and the filtered results are truncated to apredetermined bit depth. This 2-process approach is advantageous in thatthere is required no fixed order for the vertical filtering and thehorizontal filtering in the second process, and thus no signaling to adecoder is necessary regarding the order of the vertical filtering andthe horizontal filtering in the second process. However, the motioncompensation discussed in E603 requires the definition of additionalfiltering operations to generate the intermediate values. The filteringoperation applied to the intermediate values is costly and requires highcomputation complexity, in particular for video data with a high bitdepth.

Further, in the motion compensation discussed in E603, the intermediatevalues are not truncated so as to assure the accuracy of the quarter-pelsamples calculated therefrom. Thus, the bitwise precision of thecalculated values is not constant during the motion compensationdiscussed in E603. At the end of the first process explained above, theprecision of the resultant sample values is increased by an amountdetermined by the gain of the 8-tap filter. By applying the 8-tap filterto the intermediate values, the precision is then increased again by thesame amount as in the first process before truncation to a predeterminedprecision. Therefore, twice as much truncation of the resolution isneeded in the second process as is needed in the first process in orderto bring the resolution back to the original bit depth at the end of thesecond step.

SUMMARY OF THE INVENTION

In view of the above discussed prior art, the present invention providesa novel motion compensation operation for an inter-frame prediction. Themotion compensation applies a fractional sample interpolation on samplesto generate fractional samples. In the motion compensation according tothe present invention, a first fractional sample interpolation isapplied on samples in a first direction to generate a first set offractional samples. A second fractional sample interpolation is appliedon the first set of fractional samples in a second direction to generatea second set of fractional samples. In the present invention, the firstset of fractional samples is represented with a precision independent ofa precision of representation of the samples.

In an aspect of the present invention, the fractional sampleinterpolation applies an 8-tap FIR filter having one of the followingthree sets of coefficients to generate a quarter-pel sample:

[−1, 4, −10, 58, 17, −5, 1, 0];

[−1, 4, −11, 40, 40, −11, 4, −1]; and

[0, 1, −5, 17, 58, −10, 4, −1].

In an aspect of the present invention, the fractional sampleinterpolation comprises truncating each of filtered results so that thefractional samples have a desired precision. The desired precision maybe constant throughout the first and second fractional interpolationprocesses.

The truncating process may involve rounding truncated results towardsminus infinity. This is achieved by not adding any offset to thefiltered results before the truncating process.

Further in another aspect of the present invention, the samples may bescaled up to a second bit depth which is constant for the samples,regardless of the original bit depths of the samples. The second bitdepth may be set equal to a number of bits available to represent thesample, and a scaling up factor is set equal to the number of bitsavailable to represent the sample minus the original bit depth of thesample.

Further in another aspect of the present invention, the scaled upsamples are subtracted with an offset.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an exemplary hardware architecture onwhich the present invention may be implemented.

FIG. 2 is a block diagram showing a general view of a video encoder towhich the present invention may be applied.

FIG. 3 is a block diagram showing a general view of a video decoder towhich the present invention may be applied.

FIG. 4 is a block diagram showing the functional modules of an encoderaccording to an embodiment of the present invention.

FIG. 5 is a flowchart showing an encoding process performed by the videoencoder according to an embodiment of the present invention.

FIG. 6 is a flowchart showing the processes of fractional sampleinterpolation according to an embodiment of the present invention.

FIG. 7 is an illustration of filtering for fractional sampleinterpolation.

FIG. 8 is a block diagram showing the functional modules of a decoderaccording to an embodiment of the present invention.

FIG. 9 is a flowchart showing a decoding process performed by the videodecoder according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS AND THE PRESENTLY PREFERREDEMBODIMENTS

FIG. 1 shows an exemplary hardware architecture of a computer 100 onwhich the present invention may be implemented. Please note that thehardware architecture shown in FIG. 1 may be common in both a videoencoder and a video decoder which implement the embodiments of thepresent invention. The computer 100 includes a processor 101, memory102, storage device 105, and one or more input and/or output (I/O)devices 106 (or peripherals) that are communicatively coupled via alocal interface 107. The local interface 107 can be, for example, butnot limited to, one or more buses or other wired or wirelessconnections, as is known in the art.

The processor 101 is a hardware device for executing software,particularly that stored in the memory 102. The processor 101 can be anycustom made or commercially available processor, a central processingunit (CPU), an auxiliary processor among several processors associatedwith the computer 100, a semiconductor based microprocessor (in the formof a microchip or chip set), or generally any device for executingsoftware instructions.

The memory 102 comprises a computer readable medium, which can includeany one or combination of volatile memory elements (e.g., random accessmemory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and nonvolatile memoryelements (e.g., ROM, hard drive, tape, CDROM, etc.). Moreover, thememory 102 may incorporate electronic, magnetic, optical, and/or othertypes of storage media. A computer readable medium can be any means thatcan store, communicate, propagate or transport the program for use by orin connection with the instruction execution system, apparatus ordevice. Please note that the memory 102 can have a distributedarchitecture, where various components are situated remote from oneanother, but can be accessed by the processor 101.

The software 103 in the memory 102 may include one or more separateprograms, each of which contains an ordered listing of executableinstructions for implementing logical functions of the computer 100, asdescribed below. In the example of FIG. 1, the software 103 in thememory 102 defines the computer 100's video encoding or video decodingfunctionality in accordance with the present invention. In addition,although not required, it is possible for the memory 102 to contain anoperating system (O/S) 104. The operating system 104 essentiallycontrols the execution of computer programs and provides scheduling,input-output control, file and data management, memory management, andcommunication control and related services.

The storage device 105 of the computer 100 may be one of many differenttypes of storage device, including a stationary storage device orportable storage device. As an example, the storage device 105 may be amagnetic tape, disk, flash memory, volatile memory, or a differentstorage device. In addition, the storage device 105 may be a securedigital memory card or any other removable storage device 105.

The I/O devices 106 may include input devices, for example, but notlimited to a touch screen, a keyboard, mouse, scanner, microphone orother input device. Furthermore, the I/O devices 106 may also includeoutput devices, for example, but not limited to a display or otheroutput devices. The I/O devices 106 may further include devices thatcommunicate via both inputs and outputs, for instance, but not limitedto a modulator/demodulator (e.g., a modem; for accessing another device,system, or network), a radio frequency (RF), wireless or othertransceiver, a telephonic interface, a bridge, a router or other devicesthat function both as an input and an output.

As is well known by those having ordinary skill in the art, videocompression is achieved by removing redundant information in a videosequence. Many different video coding standards exist, examples of whichinclude MPEG-1, MPEG-2, MPEG-4, H.261, H.263, and H.264/AVC. It shouldbe noted that the present invention is not intended to be limited inapplication of any specific video coding standard. However, thefollowing description of the present invention is provided, using theexample of H.264/AVC standard, which is incorporated herein byreference. H.264/AVC is the newest video coding standard and achieves asignificant performance improvement over the previous coding standardssuch as MPEG-1, MPEG-2, H.261 and H.263.

In H.264/AVC, each frame or picture of a video can be broken intoseveral slices. The slices are then divided into blocks of 16×16 pixelscalled macroblocks, which can then be further divided into blocks of8×16, 16×8, 8×8, 4×8, 8×4, down to 4×4 pixels. There are five types ofslices supported by H.264/AVC. In I slices, all the macroblocks arecoded using intra-prediction. In P slices, macroblocks can be codedusing intra- or inter-prediction. P slices allow only one motioncompensated prediction (MCP) signal per macroblock to be used. In Bslices, macroblocks can be coded using intra or inter-prediction. TwoMCP signals may be used per prediction. SP slices allow P slices to beswitched between different video streams efficiently. An SI slice is anexact match for an SP slice for random access or error recovery, whileusing only intra-prediction.

FIG. 2 shows a general view of a video encoder to which the presentinvention may be applied. The blocks shown in the figure representfunctional modules realized by the processor 101 executing the software103 in the memory 102. A picture of video frame 200 is fed to a videoencoder 201. The video encoder treats the picture 200 in units ofmacroblocks 200A. Each macroblock contains several pixels of picture200. On each macroblock, a transformation into transform coefficients isperformed followed by a quantization into transform coefficient levels.Moreover, intra-prediction or inter-prediction is used, so as not toperform the coding steps directly on the pixel data but on thedifferences between pixel values and their predictions, therebyachieving small values which are more easily compressed.

For each slice, the encoder 201 generates a number of syntax elements,which form a coded version of the macroblocks of the respective slice.All residual data elements in the syntax elements, which are related tothe coding of transform coefficients, such as the transform coefficientlevels or a significance map indicating transform coefficient levelsskipped, are called residual data syntax elements. Besides theseresidual data syntax elements, the syntax elements generated by theencoder 201 contain control information syntax elements containingcontrol information as to how each macroblock has been encoded and hasto be decoded, respectively. In other words, the syntax elements aredividable into two categories. The first category, the controlinformation syntax elements, contains the elements related to amacroblock type, sub-macroblock type and information on prediction modesboth of a spatial and temporal types, as well as slice-based andmacroblock-based control information, for example. In the secondcategory, all residual data elements, such as a significance mapindicating the locations of all significant coefficients inside a blockof quantized transform coefficients and the values of the significantcoefficients, which are indicated in units of levels corresponding tothe quantization steps, are combined and become residual data syntaxelements.

The encoder 201 comprises an entropy coder which encodes syntax elementsand generates arithmetic codewords for each slice. When generating thearithmetic codewords for a slice, the entropy coder exploits statisticaldependencies among the data values of syntax elements in the videosignal bit stream. The encoder 201 outputs an encoded video signal for aslice of picture 200 to a video decoder 301 shown in FIG. 3.

FIG. 3 shows a general view of a video decoder to which the presentinvention may be applied. Likewise, the blocks shown in the figurerepresent functional modules realized by the processor 101 executing thesoftware 103 in the memory 102. The video decoder 301 receives theencoded video signal and first entropy-decodes the signal back into thesyntax elements. The decoder 301 uses the syntax elements in order toreconstruct, macroblock by macroblock and then slice after slice, thepicture samples 300A of pixels in the picture 300.

FIG. 4 shows the functional modules of the video encoder 201, which isgenerally denoted with reference numeral 400 in FIG. 4. These functionalmodules are realized by the processor 101 executing the software 103 inthe memory 102. An input video picture is a frame or a field of anatural (uncompressed) video image defined by sample points representingcomponents of original colors, such as chrominance (“chroma”) andluminance (“luma”). Other components are possible such as hue,saturation and value. The input video picture is divided intomacroblocks that each represent a square picture area consisting of16×16 pixels of the luma component of the picture color. The input videopicture is also partitioned into macroblocks that each represent 8×8pixels of each of the two chroma components of the picture color. In thegeneral encoder operation, inputted macroblocks may be temporally orspatially predicted using inter- or intra-prediction. It is, however,assumed for the purpose of discussion that the macroblocks 400 areeither all P-slice type macroblocks or all B-slice type macroblocks, ora mix of both, and subjected only to inter-prediction.

The video encoder 400 shown in FIG. 4 only transmits residuals ordifferentials between a current picture and a prediction of the currentpicture. A subtractor 410 obtains these residuals by receiving a blockof current picture to be encoded as an input signal IS and subtractingtherefrom a block of prediction picture indicated by a prediction signalPS. As illustrated in FIG. 4, the video encoder 400 includes atransform/quantization unit 420 to output quantized transformcoefficients QC. The transform/quantization unit 420 transforms theresidual from the spatial domain into the frequency domain and quantizesthe transformed residual. The video encoder 400 also includes an entropycoding unit 490 for performing entropy coding (variable length coding)of the quantized transform coefficients QC and outputting the bit streamBS.

The video encoder 400 further includes an inverse quantization/transformunit 430, and an adder 435. The inverse quantization/transform unit 430transforms the quantized transform coefficients QC back into a residualsignal RS, to which the adder 435 adds the prediction signal PS tooutput a decoded picture. A deblocking filter 437 removes blockartifacts from the decoded picture and stores the cleaned decodedpicture in a reference memory 440.

The video encoder 400 further includes a motion compensation/predictionunit 460 and a motion estimation unit 470. The motioncompensation/prediction unit 460 reads a reference picture from thereference memory 440 and applies interpolation to generate fractionalsamples at the quarter-pel positions. The motion estimation unit 470also reads the reference picture from the reference memory 440 andestimates a motion vector MV for the current picture by comparing thecurrent picture and the reference picture. The motion vector MV isoutputted to the motion compensation/prediction unit 460 and the entropycoding unit 490. The block may be divided into partitions having 4×4,4×8, 8×4, 8×8, 8×16, 16×8 or 16×16 pixels. The motion estimation unit470 performs the motion estimation on the respective partitions in theblock and outputs motion vectors MV for the respective partitions. Themotion compensation/prediction unit 460 displaces the respectivepartitions in the reference picture by the amounts and in the directionsindicated by the motion vectors to derive the prediction picture, whichis provided as the predictive signal IS to the subtractor 410 and theadder 435.

FIG. 5 is a flowchart showing a high level of encoding process performedby the video encoder 400. In Step 502, the motioncompensation/prediction unit 460 generates a prediction block. In Step504, the subtractor 410 subtracts the prediction picture from thecurrent picture to obtain a residual. In Step 506, thetransform/quantization unit 420 encodes the residual.

FIG. 6 is a flowchart showing the processes of motion compensationperformed by the motion compensation/prediction unit 460 to generate aprediction signal according to an embodiment of the present invention.In Step 602, the motion compensation/prediction unit 460 reads outreference samples from the reference memory 440. The reference memory440 stores multiple reference pictures. As discussed in Section 8.4.2and its subsections of H.264/AVC, the location of the reference samplesto be retrieved from a reference picture stored in the reference memory440 may be determined with the position of an area to be predictedwithin the current picture, a motion vector MV of the area, and areference frame index.

In Step 604, the retrieved reference samples are scaled up to increasethe precision of the reference samples. The scaling factor may bedetermined with the bit depth of the reference samples and the number ofbits available to represent the reference sample. The number of bitsavailable to represent the reference sample may be determined such that,if 16 is the maximum available number of bits for representation of thereference sample, the number of bits available to represent the samplemay be limited to 14 to prevent an overflow or underflow of data, whichmay occur during the motion compensation. If the number of bitsavailable to represent the sample is 14, the scaling factor is 6 for thereference sample which is 8 bits wide, and the retrieved referencesamples are scaled up by left-shifting their bits by 6 bits. The scalingfactor should be 4 for the reference sample which is 10 bits wide, andthe retrieved reference samples are scaled up by left-shifting theirbits by 4 bits. More generally, the scaling factor may be expressed by“the number of bits available to represent the reference sample” minus“the bit depth of the reference sample.” The scaling up processperformed in Step 604 is advantageous because it converts referencesamples stored in the reference memory in a form independent from theirbit depths and the processes following the scaling up process willbecome independent from the original bit depths of the video data.

Optionally, an offset may be subtracted from the scaled-up value of thereference sample. For example, if a signed 14-bit representation is usedto represent the scaled-up reference samples, the value of the offset tobe subtracted may be 8192 (˜2¹³), which brings the scaled-up referencesamples within the range of −8192 to 8192. Adding this offset isbeneficial for reducing the number of bits required to represent asample. Since the interpolation process described below deals withsigned sample values, a signed representation of all sample values isthus preferred. In the example above in which samples are scaled up to14-bit representations, if the offset is not subtracted, a 15-bit signedrepresentation would have to be used which is more costly than a 14-bitsigned representation obtained from subtracting the offset. Please notethat the reference samples may be scaled up and offset when they arestored in the reference memory. If the reference samples are stored inthe reference memory after scaled up and offset, Step 604 is notnecessary.

The resolution of the scaled-up reference samples is then increasedusing fractional sample interpolation to generate fractional samples atquarter-pel positions. FIG. 7 is an illustration of filtering forfractional sample motion compensation. In FIG. 7, the upper-case lettersA_(i,j) denote the scaled-up reference samples. These samples are calledinteger samples. The other samples shown by the lower-case letters, suchas a_(0,0), b_(0,0), c_(0,0), d_(0,0) . . . and r_(0,0), are fractionalsamples at quarter-pel positions. In Step 606, the quarter-pel samplea_(0,0), b_(0,0) or c_(0,0) is derived by the following interpolationequations:a _(0,0)=(−A _(−3,0)+4*A _(−2,0)−10*A _(−1,0)+58*A _(0,0)+17*A_(1,0)−5*A _(2,0) +A _(3,0))>>6b _(0,0)=(−A _(−3,0)+4*A _(−2,0)−11*A _(−1,0)+40*A _(0,0)+40*A_(1,0)−11*A _(2,0)+4*A _(3,0) −A _(4,0))>>6c _(0,0)=(−A _(−2,0)−5*A _(−1,0)+17*A _(0,0)+58*A _(1,0)−10*A _(2,0)+4*A_(3,0) −A _(4,0))>>6The fractional part of the horizontal component of the motion vectordetermines which sample is derived. For example if the fractional partis ¼, a_(0,0) is derived.

Each of the interpolation operations includes applying an 8-tap FIRfilter to the nearest integer samples and truncating the filtered resultto offset the filter gain. A sum of the coefficients of each filter isset equal to 64 (=2⁶). Thus, the truncating operation includes dividingthe filtered result by 64 or right-shifting the filtered result by 6.Please note that in the above interpolation operations, the truncatedresults are rounded towards minus infinity. By adding an offset to thefiltered results, the truncated results may be rounded towards thenearest integer. However, rounding towards minus infinity isadvantageous because it does not require adding the offset to thefiltered results. Optionally, different types of filters may be providedfrom which one filter is applied selectively, depending on a fractionaldisplacement specified by the horizontal component of a motion vector.This step may be omitted in the case where the horizontal fractionaldisplacement is zero. Please also note that there is a shift-rightoperation by 6 bits in Step 606, and a shift-left operation by 6 bits inStep 604 in the example with 8-bit inputs. Thus, Steps 604 and 606 maybe combined such that the shift-left and shift-right operations areremoved without altering the output.

In Step 608, the fractional sample d_(0,0), h_(0,0), n_(0,0), e_(0,0),i_(0,0), p_(0,0), f_(0,0), j_(0,0), q_(0,0), g_(0,0), k_(0,0) or r_(0,0)is derived by the following interpolation equations:d _(0,0)=(−A _(0,−3)+4*A _(0,−2)−10*A _(0,−1)+58*A _(0,0)+17*A_(0,1)−5*A _(0,2) +A _(0,3))>>6h _(0,0)=(−A _(0,−3)+4*A _(0,−2)−11*A _(0,−1)+40*A _(0,0)+40*A_(0,1)−11*A _(0,2)+4*A _(0,3) −A _(0,4))>>6n _(0,0)=(A _(0,−2)−5*A _(0,−1)+17*A _(0,0)+58*A _(0,1)−10*A _(0,2)+4*A_(0,3) −A _(0,4))>>6e _(0,0)=(−a _(0,−3)+4*a _(0,−2)−10*a _(0,−1)+58*a _(0,0)+17*a_(0,1)−5*a _(0,2) +a _(0,3))>>6i _(0,0)=(−a _(0,−3)+4*a _(0,−2)−11*a _(0,−1)+40*a _(0,0)+40*a_(0,1)−11*a _(0,2)+4*a _(0,3) −a _(0,4))>>6p _(0,0)=(a _(0,−2)−5*a _(0,−1)+17*a _(0,0)+58*a _(0,1)−10*a _(0,2)+4*a_(0,3) −a _(0,4))>>6f _(0,0)=(−b _(0,−3)+4*b _(0,−2)−10*b _(0,−1)+58*b _(0,0)+17*b_(0,1)−5*b _(0,2) +b _(0,3))>>6j _(0,0)=(−b _(0,−3)+4*b _(0,−2)−11*b _(0,−1)+40*b _(0,0)+40*b_(0,1)−11*b _(0,2)+4*b _(0,3) −b _(0,4))>>6q _(0,0)=(b _(0,−2)−5*b _(0,−1)+17*b _(0,0)+58*b _(0,1)−10*b _(0,2)+4*b_(0,3) −b _(0,4))>>6g _(0,0)=(−c _(0,−3)+4*c _(0,−2)−10*c _(0,−1)+58*c _(0,0)+17*c_(0,1)−5*c _(0,2) +c _(0,3))>>6k _(0,0)=(−c _(0,−3)+4*c _(0,−2)−11*c _(0,−1)+40*c _(0,0)+40*c_(0,1)−11*c _(0,2)+4*c _(0,3) −c _(0,4))>>6r _(0,0)=(c _(0,−2)−5*c _(0,−1)+17*c _(0,0)+58*c _(0,1)−10*c _(0,2)+4*c_(0,3) −c _(0,4))>>6The fractional parts of the horizontal and vertical components of themotion vector determine which sample is derived. For example if thehorizontal fractional part is ¼ and the vertical fractional part is ¾,p_(0,0), is derived.

The above interpolation operations include applying 8-tap FIR filtersvertically to the samples A_(0,j), a_(0,i), b_(0,i) and c_(0,i), wherei=−3.4, and truncating the filtered results by a factor of 6. Pleasenote that in the present invention, every filtered result is truncatedand that no filtered result is used as an untruncated intermediate valueto calculate fractional samples therefrom. In the present invention,therefore, the precision of the samples values is constant throughoutthe fractional sample interpolation. Neither step 606 nor step 608alters the precision in the representation of samples.

In the above embodiment of fractional sample interpolation, the sets offilters used in the vertical and horizontal filtering operations are all8-tap filters. It should be noted that filters with different tapnumbers may be used for the fractional motion interpolation. Forexample, it is desirable to use filters with fewer taps for smallerblock sizes. The use of filters with fewer taps reduces the number ofsamples to be retrieved from the reference memory, whereby the memorybandwidth can be reduced. The use of filters with fewer taps alsoreduces the number of multiplications in the filtering operations andimproves the efficiency of motion compensation. H.264/AVC allows amacroblock or sub-macroblock to be divided into one or more partitionsfor the purpose of motion compensation. In H.264/AVC, a macroblockhaving a size of 16×16 may be divided into one or more partitions eachhaving a size of 4×4, 4×8, 8×4, 8×8, 8×16 or 16×8. Accordingly, filterswith a different number of taps may be prepared and applied selectivelyaccording to the width of a partition. For example, an 8-tap filter maybe applied to partitions having a width equal to or larger than 8, and a4-tap filter may be applied to partitions having a width smaller than 8.In one embodiment, such a 4-tap filter has coefficients [−4 36 36−4].JCT-VC Draft E603 has introduced the concept of a coding unit (CU). A CUhaving a size of N×N may be divided into one or more partitions having asize of N×N, N×N/2 or N/2×N. Filters with different tap numbers may beapplied selectively whether the height or width of a CU is N or N/2.

In the above embodiment of fractional sample motion compensation, thehorizontal fractional sample interpolation (Step 606) precedes thevertical fractional sample interpolation (Step 608). It should be notedthat the order may be switched as long as the order is agreed upon inadvance between an encoder and a decoder. In another embodiment, theorder may be signaled from an encoder to a decoder. When the referencessamples are 8 bits wide, the result of calculation will be the sameirrespective of the order of horizontal and vertical interpolation. Ifit is the case, no prior agreement is needed between an encoder and adecoder regarding the order.

In the motion compensation discussed in JCT-VC Draft E603, the samefractional samples can be derived whether the intermediate values arederived vertically and applied horizontally or derived horizontally andapplied vertically. This is beneficial in a sense that there is no prioragreement is necessary between an encoder and a decoder regardingwhether the intermediate values should be derived vertically and appliedhorizontally or derived horizontally and applied vertically. However,the intermediate values have higher precision than in the presentinvention and the cost of applying a second FIR filter to theseintermediate values is higher than that of the present invention.Generally, the cost increases in proportion to the precision. This costmay be significantly higher in software where the precision exceeds acommon word size such as 16 bits. In such a case the precision isfurther extended to match the next common word size. For example,intermediate values with 17 bits of precision would be extended to 32bits. The higher precision of intermediate values as discussed in JCT-VCDraft E603 does have a significant cost impact, without providing asignificant benefit in terms of accuracy of the prediction signal.

In the embodiment of the present invention described above, the order ofthe horizontal and vertical interpolations needs to be fixed between anencoder and a decoder. However, fixing the order of the horizontal andvertical interpolations will not cause any cost to the encoder and thedecoder. Rather, by fixing the order, the motion compensation canoperate at a lower, yet sufficiently high, precision to lower the costof operations such as multiplications. Also, by fixing the order, thecalculation precision becomes constant throughout the fractional sampleinterpolation. The filtering calculation can thus be simplified in thatall the multiplications between samples and coefficients can beperformed within fixed bit ranges, for example, a 16-bit signed rangefor samples and an 8-bit singed range for filter coefficients. Since allthe multiplications are performed within fixed bit ranges, the filteringoperations can efficiently be implemented in parallel on SIMDarchitectures.

Turning back to FIG. 6, a transformation is applied to the derivedsamples in Step 610. An example of such a transformation is described asa weighted sample prediction process in Section 8.4.2.3 and itssubsections of H.264/AVC. The weighted sample prediction processcombines two sets of samples by a linear interpolation. Thetransformation can improve coding efficiency for scenes containingfades. The value of the combined samples resulting from thetransformation may be clipped to fit within a specified range, e.g.16-bit signed range. Please note that Step 610 can be omitted where nosuch transformation is necessary.

It is then checked in Step 612 whether another prediction needs to begenerated for motion compensation. For example, when bidirectionalaveraging is to be performed, two predictions need to be generated foraveraging thereof. If it is determined that another prediction isneeded, the process returns to Step 602. In Step 614, two predictionsare averaged to generate a bidirectional prediction. If bidirectionalaveraging is not instructed, Step 614 is skipped.

In Step 616, a generated prediction is scaled down and clipped to avalid range. The scaling down operation performed in Step 616 undoes thescaling up operation performed in Step 604 and right-shifts thegenerated prediction by the same number of bits as used to left-shiftthe samples in Step 604. It is preferable to add a rounding offsetbefore the right-shift operation to round the scaled down value towardsthe nearest integer. If the offset is subtracted from a sample in Step604, the same offset is added to the prediction. The valid range towhich the prediction is clipped is typically defined as {0 . . . 255},where the stored reference samples are 8 bits wide. Generally, the validrange may be defined as {0 . . . 2^(M)−1} for reference samples whichare M bits wide.

In a practical implementation of the present invention, the steps neednot be performed separately as shown in FIG. 6, and some steps may becombined. For example, shifting operations performed separately in stepsof FIG. 6 may be combined. An example is that the right-shift operationincluded in the bidirectional averaging performed in Step 614 and theright-shift operation included in scaling down performed in Step 616 maybe combined. Further, multiple functions or circuits may be implementedfor circumstances where processes can be simplified, for example, wherethere are no fractional displacements and thus FIR filtering is notrequired.

FIG. 8 shows the functional modules of the video decoder 301, which isgenerally denoted with reference numeral 800 in FIG. 8. These functionalmodules are realized by the processor 101 executing the software 103 inthe memory 102. In FIG. 8, an entropy decoding unit 810 obtains a bitstream BS outputted by the moving picture coding apparatus 400 andperforms entropy decoding processing of the bit stream. The entropydecoding unit 810 then outputs the quantized transform coefficients QCto the inverse quantization/inverse transform unit 820. The entropydecoding unit 810 also outputs the motion vector MV to a motioncompensation/prediction unit 860. The inverse quantization/inversetransform unit 820 performs decoding processing of inverse quantizationand inverse DCT transform on the quantized transform coefficients QC.The inverse quantization/inverse transform unit 820 then outputs theresidual signal RS.

An adder 830 adds the residual signal RS to the prediction signal PSobtained from the motion compensation unit 860 and outputs a decodedpicture to a deblocking filter 840. The deblocking filter 840 eliminatesblock artifacts from the decoded picture and stores it in a referencememory 850. The decoded picture is retrieved from the memory 850 andoutputted as an output signal OS.

The motion compensation/prediction unit 860 retrieves samples in areference picture stored in the reference memory 850 and performs thefractional sample interpolation as shown in FIG. 7. This motioncompensation/prediction unit 860 generates a prediction based on themotion vector MV and the reference picture on which the fractionalsample interpolation has been performed. In this way, the motioncompensation/prediction unit 860 outputs the prediction signal PS to theadder 830.

FIG. 9 is a flowchart showing a high level of decoding process performedby the video decoder 800. In Step 902, the motioncompensation/prediction unit 860 generates a prediction block. In Step904, the inverse quantization/inverse transform unit 820 decodes thequantized transform coefficients QC and outputs the residual signal RS.In Step 906, the adder 830 adds the residual signal RS to the predictionsignal PS.

Whereas many alterations and modifications of the present inventionwill, no doubt, become apparent to a person of ordinary skill in the artafter having read the foregoing description, it is to be understood thatany particular embodiment shown and described by way of illustration isin no way intended to be considered limiting. Therefore, references todetails of various embodiments are not intended to limit the scope ofthe claims, which in themselves recite only those features regarded asessential to the invention.

The invention claimed is:
 1. A video decoding method for motioncompensation performed under an inter-frame prediction to decode atarget picture, the method comprising computer executable steps executedby a processor of a video decoder to implement: decoding a residual anda motion vector received from an encoder; referencing to the motionvector to retrieve a reference sample from a reference picture stored ina reference picture memory, wherein the reference picture stored in thereference picture memory and the reference sample retrieved from thereference picture are both represented with a first bit depth;performing a scaling-up operation and a first fractional sampleinterpolation in a first direction on the retrieved reference sample togenerate a first set of fractional samples represented with a second bitdepth to which the first bit depth is scaled up by a scaling-up factor,wherein the second bit depth is constant and set equal to a number ofbits available to represent the fractional sample, and the scaling-upfactor is set equal to the second bit depth minus the first bit depthand is variable to keep the second bit depth constant and independentfrom a change of the first bit depth; performing a second fractionalsample interpolation on the first set of fractional samples in a seconddirection to generate a second set of fractional samples representedwith the second bit depth; referencing fractional parts of the motionvector to derive a prediction sample from the first and second sets offractional samples, wherein the prediction sample is represented withthe second bit depth; scaling down and clipping the prediction sample togenerate a prediction picture scaled down from the second bit depth tothe first bit depth; and adding the prediction picture and the residualto reconstruct the target picture represented with the first bit depth.2. The method according to claim 1, wherein the fractional sampleinterpolation applies an 8-tap FIR (Finite Impulse Response) filterhaving one of the following three sets of coefficients to generate aquarter-pel sample: [−1, 4, −10, 58, 17, −5, 1, 0]; [−1, 4, −11, 40, 40,−11, 4, −1]; and [0, 1, −5, 17, 58, −10, 4, −1].
 3. The method accordingto claim 2, wherein the fractional sample interpolation comprisestruncating each filtered result so that the fractional samples arerepresented with the second bit depth.
 4. The method according to claim3, wherein the second bit depth is maintained throughout the first andsecond fractional interpolation processes.
 5. The method according toclaim 3, wherein truncating each filtered result comprises roundingtruncated results towards minus infinity.
 6. The method according toclaim 1, further comprising subtracting an offset from the scaled upsamples.
 7. A video encoding method for motion compensation performedunder an inter-frame prediction, the method comprising computerexecutable steps executed by a processor of a video encoder toimplement: comparing a target picture and a reference picture stored ina reference picture memory to estimate a motion vector, wherein thereference picture stored in the picture memory is represented with afirst bit depth; referencing the motion vector to retrieve a referencesample from the reference picture, wherein the reference sampleretrieved from the reference picture is represented with the first bitdepth; performing a scaling-up operation and a first fractional sampleinterpolation in a first direction on the retrieved reference sample togenerate a first set of fractional samples represented with a second bitdepth to which the first bit depth is scaled up by a scaling-up factor,wherein the second bit depth is constant and set equal to a number ofbits available to represent the fractional sample, and the scaling-upfactor is set equal to the second bit depth minus the first bit depthand is variable to keep the second bit depth constant and independentfrom a change of the first bit depth; performing a second fractionalsample interpolation on the first set of fractional samples in a seconddirection to generate a second set of fractional samples representedwith the second bit depth; referencing fractional parts of the motionvectors to derive a prediction sample from the first and second sets offractional samples, wherein the prediction sample is represented withthe second bit depth; scaling down and clipping the prediction sample togenerate a prediction picture scaled down from the second bit depth tothe first bit depth; and subtracting the prediction picture from thetarget picture to derive a residual for encoding.
 8. The methodaccording to claim 7, wherein the fractional sample interpolationapplies an 8-tap FIR (Finite Impulse Response) filter having one of thefollowing three sets of coefficients to generate a quarter-pel sample:[−1, 4, −10, 58, 17, −5, 1, 0]; [−1, 4, −11, 40, 40, −11, 4, −1]; and[0, 1, −5, 17, 58, −10, 4, −1].
 9. The method according to claim 8,wherein the fractional sample interpolation comprises truncating eachfiltered result so that the fractional samples are represented with thesecond bit depth.
 10. The method according to claim 9, wherein thesecond bit depth is maintained throughout the first and secondinterpolation processes.
 11. The method according to claim 9, whereintruncating each filtered result comprises rounding truncated resultstowards minus infinity.
 12. The method according to claim 7, furthercomprising subtracting an offset from the scaled up samples.